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AUTOMATIC PANNING OF DIGITAL CONTENT WHILE ZOOMED 



Field Of The Invention 



The invention relates generally to digital video application and more particularly 
to automatic panning of digital content while zoomed. 



Digital televisions receive and display images that are broadcast as digital signals, 
for example as a stream of MPEG2 data. The digital televisions receive the digital 
signals on a digital medium, and place the MPEG2 data into a video buffer, much as a 
computer receives and stores video or graphics information. 
Digital televisions may include many of the features of prior generations of analog 
televisions, including the ability to "zoom." Zooming allows a person watching 
television to magnify a selected portion of the television screen. The selected portion is a 



"zoom frame." The television displays the zoom frame, magnifying it to cover the entire 
screen, rather than the entire MPEG2 frames (i.e., the "full frames") as received by the 
television. 

Internally, the television defines the zoom frame in terms of its position and size 
within the full frame. Unfortunately, as persons and things move about the screen, they 
enter and leave the selected portion defined as the zoom frame. For example, if a the full 
frame belongs to a motion picture in which a person walks from the left edge of the 
screen to the right edge of the screen, the person will walk through the zoom frame. 
Panning of the camera exacerbates the problem. Panning is a cinematographic technique 
in which a cameraman pivots the camera to show a scene that is too large to be shown 
conveniently in a single frame. When a camera is panned, the image appears to slide 



Background Of The Invention 




from one edge of the full frame to the other edge of the full frame. Accordingly, unless 
the zoom portion is moved with respect to the full frame, the zoom portion includes 
images that slide from one edge of the zoom frame to the other. 

Brief Description Of The Drawings 

FIG. 1 shows a digital television system that includes a video capture module 12, 
an MPEG2 video encoder 14, digital audio/video storage 20, an MPEG2 Decoder 60, a 
zoom module 70, and a display 88. 

FIGS. 1A and IB depict alternative embodiments that utilize zoom modules 210 
and 308, respectively. 

FIG. 2 shows a full frame 102 and a zoom frame 112 that are examples of one 
situation that might arise during operation of the digital television system of FIG. 1. 

FIG. 3 shows a method in accordance with one embodiment of the present 
invention. 

Detailed Description Of a Preferred Embodiment of The Invention 

The present invention includes, among various aspects, a digital television system 
that has a zoom module. When the digital television system is in a zoom mode, the zoom 
module receives a full frame, and displays zoom frame that includes only a portion of the 
full frame. The zoom module determines a relationship between the zoom frame and the 
full frame. The zoom module also identifies an object within a zoom frame, and a motion 
vector of the object with respect to a background of the zoom frame. As the object 
moves within the zoom frame, the zoom module adjusts relationship between the zoom 
frame and the full frame so that the object remains within the zoom frame. 

^ Various aspects of the present invention may be more fully understood with 
reference to FIGs. 1-3. FIG. 1 shows a /igital television system that includes a video 
capture module 12, an MPEG2 video /ncoder 14, digital audio/video storage 20, an 
MPEG2 Decoder 60, a zoom module 70, and a display 88. The video capture module 12 



is, for example, a tuner that receives videc/inputs 34 and processes them to produce 
digitized video data 36. The video capture module 12 also captures audio inputs 44 and 
processes them to produce digitized/audio data 46. For example, the video capture 
module 12 processes video inputs from NTSC broadcasts, VCR outputs, etc. and 
converting them into digitized/video data 36. The audio capture module 16, which 
produces digitized audio dara, 42 therefrom, receives an audio input 40. The video inputs 
34 and audio inputs 44 a^e part of a program input that may be originated from a 
television broadcast, oable broadcast, satellite broadcast, VCR output, DVD output, or 
any audio/video analog signal. 

The MPEG2 video encoder 14 receives the digitized video data 36 and encodes it 
into MPEG2 encoded video data 38. In addition to processing video data, audio data is 
processed. The MPEG audio encoder 18 converts the digitized audio data 42 into 
encoded audio data 44. The video capture module 12 may further optionally include one 
or more of a close-captioned encoder, a parental control encoder, and a copy protection 
encoder. Such a conversion is utilized in the All-in- Wonder product produced and 
manufactured by ATI Technologies. 

The digital audio/video storage 20 stores the MPEG encoded video data 38 and 
the MPEG encoded audio data 44. The digital audio/video storage 20 contains a file 
management in a processing module (not shown) and a memory that provide write 
control information to the digital audio/video storage 20. The processing module may be 
a CPU (central processing unit) such as a stand-alone microprocessor, micro-controller, 
digital signal processor (DSP), central processing unit (CPU), a coprocessor, or any 
device that manipulates digital data based on operational instructions. The memory may 
be random access memory, cache memory, floppy disk memory, hard drive or hard disk 
memory, floppy drive or hard disk memory, zip drive memory, magnetic tape memory, 
DVD-RAM memory, and/or any device that stores digital information which can be read 
from and written to by the CPU. 
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It will be recognized that the digital audicf/video storage 20 is optional, and that in 
a second embodiment the MPEG encoded audfo data 44 and the MPEG encoded video 
data 38 may be provided directly to the digipil video retrieval module 22. A dashed 
arrow on FIG. 1 represents this data path,Aom the MPEG2 video encoder 14 and the 
MPEG audio encoder 18 to the to the digital video retrieval module 22. When the MPEG 
encoded audio data 44 and the MPEGr encoded video data 38 is provided directly to the 
digital video retrieval module 22, me MPEG encoded audio data 44 and the MPEG 
encoded video data 38 compris^ a live multimedia display output. 

In a third embodiment, the digital audio/video storage 20 is implemented in 
parallel with the Zoom module 70, and both the digital audio/video storage 20 and the 
Zoom module 70 receive the MPEG encoded audio data 44 and the MPEG encoded video 
data 38 may be provided directly to the Zoom module 70. When included, the digital 
audio/video storage 20 may be a portion of a file management system that stores the 
MPEG encoded audio data 44 and the MPEG encoded video data 38 in a storage medium 
or other an archiving module. The storage medium may be a hard drive of a personal 
computer, RAM of a personal computer, floppy disk, or any particular digital storage 
medium. 



The digital video retrieval module 22 receives the MPEG2 encoded video data 38 
and the MPEG encoded audio data 44, and provides the MPEG2 encoded video data 38 
and the MPEG encoded audio data 44 to the zoom module 70. The digital video retrieval 
module 22 is, for example, an MPEG2 video/audio decoder that includes a processing 
module 30 and memory 32. 



The processing module 30 may be a CPU (central processing unit) such as a 
stand-alone microprocessor, micro-controller, digital signal processor (DSP), central 
processing unit (CPU), a coprocessor, or any device that manipulates digital data based 
on operational instructions. The memory may be read only memory, random access 
memory, cache memory, floppy disk memory, hard drive or hard disk memory, floppy 
drive or hard disk memory, zip drive memory, magnetic tape memory, DVD memory, 
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and/or any device that stores digital information which can be read from and written to by 
the CPU. 

The digital video retrieval module 22 is operably coupled to receive the encoded 
video and produce a video component of the decoded program. The digital video 
retrieval module 22 may utilize an MPEG2 decoding scheme such that the encoded video 
is stored in an MPEG2 format and is decoded into an uncompressed video component. 
The zoom module 70 receives video data signals 52 from the digital video retrieval 
module 22. The video data signals 52 comprise a stream of video images, or video 
frames, that are presented in sequence on the display 88. The zoom module 70 presents 
the image on the display 88. 

The video component is provided to a display 88. The display 88 may be a 
television, a monitor, a CRT, and/or a LCD display, and further includes one or more 
speakers that may include a preamplifier stage to amplify an audio component to provide 
the appropriate volume levels. 

The elements of Figure 1 may be implemented as separate processing modules 
having associated memories or as a single processing module having an associated 
memory. Such a processing module may be a single processing device or a plurality of 
processing devices. A processing device may be a microprocessor, a microcontroller, 
digital signal processor, state machine, logic circuitry, and/or any device that manipulates 
signals (analog or digital) based on operational instructions. The associated memory(ies) 
stores the operational instructions executed by the corresponding processing module(s). 
Such a memory may be a single memory device or a plurality of memory devices. Such a 
memory device may be a random access memory, read-only memory, floppy disk 
memory, hard drive memory, and/or any device that stores digital information. 

FIG. 1 depicts time shifting a picture by converting an analog television picture to 
a digital picture. However, the present invention is not limited to only systems that 
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convert analog television pictures to digital pictures. The present invention is equally 
applicable to all digital systems such as cable set-top box systems and DVD players. 

FIG. 1 A is an alternative embodiment in which a digital television receiver 200 
receives digital television signals from one of a dish antenna 202 or a cable DVD set-top 
box 204. The digital television receiver 200 is connected to a video decoder 208. The 
zoom module 210 is connected to receive the signals from the video decoder 208. As 
described above the zoom module 210 provides images to be displayed to the display 
210. 

FIG. IB is a anther alternative embodiment in which a disc reader 300 receives 
image signals from one of a computer hard drive 302 or a DVD player 304. The disc 
reader 300 sends digital signals to a digital video decoder 306, which decodes the digital 
signals. A zoom module 308 is connected to receive signals from the video decoder 306. 
As described above the zoom module 308 provides images to be displayed to the display 
310. 

FIG. 2 shows a full frame 102 and a zoom frame 112 that are examples of one 
situation that might arise during operation of the digital television system of FIG. 1. As 
stated previously, the video capture module 12 processes video inputs from NTSC 
broadcasts, VCR outputs, etc. and converting them into digitized video data 36. In the 
example shown in FIG. 2, the video inputs depict several automobiles 104-108 driving 
past a crowd 1 10 of onlookers. 

It will be understood that the full frame 102 is merely a single frame of a 
sequence of frames that collectively comprise the digitized video data 36. The video 
inputs may, for example, be photographed by a cameraman who attempts to keep all of 
the automobiles 104-108 within the full frame 102. It will be understood that 
"photographed" may be interpreted as filmed, videotaped, or transmitted as a moving 
video image. 
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The cameraman may be located in a fixed position, and therefore "pans" the 
camera as the automobiles 104-108 move past the crowd. To pan the camera, the 
cameraman rotates or pivots the camera as the objects, in this case the automobiles 104- 
108, move. Because the cameraman does not have any particular interest in, for example, 
automobile 104, and merely intends to position or pivot the camera such that all the 
automobiles 104-108 remain within the frame, the cameraman may allow any particular 
automobile (such as automobile 104) to be located near an edge of the full frame 102. 
This may be particularly necessary as the angle of view between the automobile 104 and, 
for example, automobile 108, increases. The automobile 104 may be located near the left 
edge of the full frame 102, and the automobile 108 may be located near the right edge of 
the full frame 102. 

The full image 102 is shown on the display 88, and fills the entire display 88. 
Accordingly, a user viewing the display 88 sees the automobiles 104-108 on the display 
88. However, the user may be particularly interested in automobile 104. In other words, 
although the cameraman has photographed the automobiles 104-108, the user desires to 
"zoom" in on automobile 104. 

Zooming is a well-known concept in video processing. A portion of the full 
image 102 is identified and enlarged to fill the entire display 88. The user can identify 
the portion of the full image 102 that is of particular interest via an input device such as a 
remote control, a keyboard, a mouse, a light pen, a laser pointer, a or touch screen. 
The user may select the portion of the display 88 by using the remote control. For 
example, the user may press "zoom" button on the remote control and then use a zoom- 
control portion of the remote control to select a zoom portion of the display 88. 
Alternately, the user may use a numeric portion of the remote control to indicate a zoom 
portion of the screen. Methods may be devised to designate the area to zoom using a 
standard remote control. 

Another method of zooming is for the user to use a single button on the remote 
control to select the portion of the display 88 having a center coincident with the center of 
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the display 88, such that the portion has zoom percentage programmed by the user via the 
remote control. For example, if the user enters a zoom percentage of 200%, the zoom 
module 70 determines that the user has selected a portion of the screen having a center 
coincident with the center of the display 88 and having a size equal to 50% (i.e., the 
reciprocal of 200%) of the display 88. 

The user may also select from among a pre-defined set of portions of the display 
88, such as a particular quadrant of the display 88. For example, a dedicated key on the 
remote control may allow the user to zoom onto a portion of the screen having a center 
coincident with the center of the display 88 and having a size equal to 50% (i.e., the 
reciprocal of 200%) of the display 88. 

The zoom portion may be defined by three components: a horizontal position 
within the full frame, a vertical position within the full frame, and a size percentage 
(zoom factor) with respect to the full frame. For example, the position may be a distance 
from the left edge of the full frame and a distance from the top edge of the full frame. 
Alternatively, the zoom portion may be defined by a horizontal position and vertical 
position of a corner of the zoom frame 112 within the full frame 102, and a horizontal 
position and vertical position of an opposite corner of the zoom frame 1 12 within the full 
frame 102. 

The zoom portion is initially represented on the display 88 as a transparent frame 
having visible edges. Regardless of how the zoom portion is defined, the user may then 
adjust the zoom portion to direct the zoom portion by altering its size and location on the 
full frame 102 shown on the display 88. In other words, although the full frame 102 is 
under the control of the cameraman, the zoom portion 112 within the full frame 102 is 
under the control of the user. 

Once the user has selected a zoom portion 102, the zoom module 70 switches to a 
zoom mode. When in the zoom mode, the zoom module 70 displays only the zoom 
portion 1 12 on the display 88, enlarged as necessary to fill the display 88. 



As the user continues to view the zoom portion 1 12, however, the automobile 104 may 
move within the zoom portion 112. For example, if the cameraman redirects the camera 
to show the automobiles 104 and 106, and does not show automobile 108, then 
automobile 104 may be located closer to the center of the full frame 102. This is show in 
the second drawing of FIG. 2. However, as the automobile 104 moves within the full 
frame 102, the automobile 104 may no longer be completely contained within the zoom 
frame 112. 

In accordance with one embodiment of the present invention, the zoom module 70 
includes an MPEG2 motion detector that determines motion vectors within the zoom 
portion of tlie display 88. The MPEG2 motion detector decomposes each frame of the 
video stream into a plurality of discrete elements. The image elements are objects or 
persons shown in the image. For example, the image elements may include the 
automobile 104, the automobile 106, the automobile 108, and the crowd 110. 
Once the user has selected a zoom portion of the display 88, the zoom module 70 
determines a compensated motion vector for each of the image elements in the zoom 
frame. The compensated motion vector, however, is not a motion vector as retrieved 
from the digital video retrieval module 22. Instead, the compensated motion vector is 
determined from a comparison of the zoom portion of the image with the full image as 
received from digital video retrieval module 22. 

As shown in FIG. 2, the automobile 104 moves with respect to the full frame 102. 
The automobile 102 occupies a substantial portion of the zoom frame 112. Among all of 
the objects shown in FIG. 2, the automobile 104 has the largest motion vector. 
Accordingly, the motion vector associated with the automobile 104 is larger than the 
motion vector associated with any other object within the zoom frame 112. In 
accordance with one embodiment of the present invention, the zoom module 70 detects 
that the size, location, and motion vector of the automobile 104 will soon cause a portion 
of the automobile 104 to extend beyond the zoom portion 112. Accordingly, in 
accordance with one embodiment of the present invention, the zoom module 70 adjusts 
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the zoom portion 1 12 such that the automobile 104 does not extend beyond the zoom 
portion 112. 

The zoom module 70 performs this adjustment, in this example, by adjusting the 
horizontal position within the full frame, the vertical position within the full frame, and 
the size percentage (zoom factor) with respect to the full frame, or by adjusting the 
horizontal position and vertical position of a corner of the zoom frame 112 within the full 
frame 102 and the horizontal position and vertical position of the opposite corner of the 
zoom frame 112 within the full frame 102. The zoom module 70 also adjusts the zoom 
factor, where the zoom factor is a ratio of the area included within the zoom frame 1 12 
and the area included within the full frame 102. The adjustment can be made 
continuously, as the automobile 104 moves within the zoom frame 1 12, or can occur 
abruptly as an edge of the automobile 104 approaches an edge of the zoom frame 112. 

Moreover, in accordance with one embodiment of the present invention, as the 
camera pans past the crowd, the zoom module 70 determines a motion vector for objects 
within the full frame 102. The objects within the full frame 102 include objects within 
the zoom frame 112, but also include objects that are not within the zoom frame 112 and 
objects that are only partially within the zoom frame 112. The motion vectors of the 
objects within the full frame 102 are used to determine whether the camera is panning. 
For example, without panning, the automobiles 104-108 would have a large motion 
vector and would appear to move from left to right past the crowd 110, and the crowd 
110 would have a very small motion vector. However, with panning (i.e., pivoting or 
other motion of the camera) to follow the automobiles 104-108, the automobiles 104-108 
have a small motion vector, and the crowd 110 appears to have a large motion vector in 
the opposite direction. It is to be understood that the above described method uses 
motion vectors and MPEG2 has motion vectors built in. However, for non-MPEG2 
content the motion vectors must be created. 

It is desirable that the zoom frame 1 12 be adjusted to follow the automobile 104, 
and not be adjusted to follow the crowd 110. However, with panning, the crowd 110 had 
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a larger motion vector than the automobile 104. Accordingly, when determining the 
compensated motion vectors for objects within the zoom frame 1 12, the zoom module 70 
"subtracts" a motion vector of the full frame 102 taken as a whole. In other words, the 
zoom module 70 determines that a object (i.e., the crowd 110) has a large motion vector 
in one direction when observed in the full frame 102, and yet has a small motion vector 
when observed in the zoom frame 112. Accordingly, the zoom module 70 determines 
that the camera is panning past the crowd 1 10, and the crowd 1 10 is not moving past the 
camera. 



It may happen that the adjustment of the zoom frame 1 12 itself would cause a 
portion of the zoom frame 1 12 to extend beyond an edge of the full frame 102. This is 
shown in the fourth picture of FIG. 2, in which the automobile 104 passes the automobile 
106, and approaches the edge of the full frame 102. For example, the cameraman may 
slow the panning to follow the automobile 106, while the user may continue to have a 
particular interest in the automobile 104. In accordance with one embodiment of the 
present invention, when this occurs, the zoom module 70 cancels the zoom mode and 
returns to displaying the full frame. 

fi^^-^FlG. 3 shows a method in accordance witl^ne embodiment of the present 
invention. The method begins at step 116, when^he zoom module 70 determines to 
begin a zoom mode. The zoom mode may be /ntered automatically, or in response to 
user input. At step 120, the zoom module 7y determines a zoom portion of the full 
image. The zoom portion may be defined three components: a horizontal position 
within the full frame, a vertical position within the full frame, and a size percentage 
(zoom factor) with respect to the full frame. For example, the position may be a distance 
from the left edge of the full frame aj(d a distance from the top edge of the full frame. 
Alternatively, the zoom portion ma/y be defined by a horizontal position and vertical 
position of a corner of the zoom irame within the full frame, and a horizontal position 
and vertical position of an opposite corner of the zoom frame within the full frame. 
At step 130, the zoom moduli displays the selected portion (i.e., the zoom portion) on the 
display. At step 140, the zoom module detects motion of an object within the portion of 
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the image. For example, the zoom module detects the^(5mpensated motion vector 
associated with the automobile 104 within the zopm frame. This optionally includes 
subtraction of a motion vector associated wjtffme background. At step 150, the zoom 
module selects a second portion of the i^fage. It will be recalled that a video image is 
actually a sequence of still framespresented rapidly so as to create an appearance of 
motion. Accordingly, the secpfid portion of the image is simply a next zoom frame in a 
sequence. The second ppraon may have the same size and location with respect to the 
full frame as the fir^zoom frame, or may be adjusted with respect to the first zoom 
frame. If the s^iond portion is adjusted continuously in response to the compensated 
motion vector of an object within the first zoom frame. If desired, the adjustments can be 
accumulated and delayed until the object nears an edge of the second portion of the 
inyfge. 

At step 160, the zoom module determines whether at least one edge of the second 
portion of the image extends beyond the full frame. Such a situation is shown in the 
fourth picture of FIG. 2. If at least one edge of the second portion of the image extends 
beyond the full frame, then the zoom module cancels the zoom mode. 
Optionally, at step 170, the zoom module determines whether a difference between the 
first portion of the image and the second portion of the image exceeds a predetermined 
threshold. In other words, the zoom module determines whether the zoom frame is 
moving too quickly across the full frame. If the zoom frame is moving too quickly across 
the full frame, then the zoom module cancels the zoom mode. Otherwise, the zoom 
module returns to step 130. 

The method continues until interrupted the zoom state is terminated or there is no 
content being received. Optionally, the zoom state may be cancelled automatically by 
changes within the content itself, by commands from the user, or by various 
measurements with respect to the zoom frame exceeding various thresholds associated 
with the full frame. 
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It should be understood that the implementation of other variations and 
modifications of the invention in its various aspects will be apparent to those of ordinary 
skill in the art, and that the invention is not limited by the specific embodiments 
described. For example, the zoom module 70 may keep track of the motion of several 
objects, so that rather than canceling the zoom mode the zoom module 70 can instead 
switch to tracking a different object. It is therefore contemplated to cover by the present 
invention, any and all modifications, variations, or equivalents that fall within the spirit 
and scope of the basic underlying principles disclosed and claimed herein. 
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